https://app.knowbase.one/vault/rfc/docid/548
RFC9309: Robots Exclusion Protocol
Internet Engineering Task Force (IETF) M. Koster Request for Comments: 9309 Category: Standards Track G. Illyes ISSN: 2070-1721 H. Zeller ...